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ABSTRACT 



The Employment and Training Administration’s (ETA's) 
development and implementation of America’s Career Kit (ACK) , which is an 
online career development resource for individuals needing job search 
assistance, career guidance, salary data, and training and educational 
resources, was evaluated. The evaluation was designed to determine whether 
ACK has developed adequate performance measures to gauge the program’s 
overall success and addressed the compliance requirements of the Department 
of Labor’s (DOL’s) equal employment opportunity agencies. Several concerns 
related to development, implementation, and monitoring of ACK were 
identified. All the identified issues were related to the America’s Job Bank 
(AJB) component of ACK., The evaluation established that AJB’s overall 
effectiveness ' is difficult to assess and that ETA has not fully addressed the 
compliance concerns of DOL agencies responsible for equal employment 
regulations. The following were among the nine recommendations for improving 
ACK: (1) conduct an online survey of AJB’s users to identify factors 

contributing to use of the site; (2) gather customer satisfaction data 
regarding ACK/AJB customer service; (3) take a proactive approach to meeting 
DOL regulatory requirements regarding data collection and impact t analysis; 

(4) provide access to AJB for limited English-proficient job seekers; and (5) 
incorporate the subcategory of Campaign Veterans into AJB’s self- 
identification section. (ETA’s complete response is appended.) (MN) 
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ACRONYMS 
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- 


America’s Career Kit 
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- 


America’s Job Bank 
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- 
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- 


Civil Rights Center 


DOL 


- 


U.S. Department of Labor 


EEO 


- 


Equal Employment Opportunity 


EO 


- 


Executive Order 
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- 


Employment Standards Administration 


ETA 


- 


Employment and Training Administration 


FY 


- 


Fiscal Year 
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Office of Inspector General 
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- 
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- 


Program Y ear 
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- 


Veterans’ Employment and Training Service 
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EXECUTIVE SUMMARY 



We conducted an evaluation of the Employment and Training Administration’s (ETA) 
development and implementation of America’s Career Kit (ACK). Our evaluation covered 
Program Years (PY) 1997 through 2000. The evaluation was designed to assess whether ACK 
has: (1) developed adequate performance measures to gauge the overall success of the program, 
and (2) addressed the compliance requirements of Department of Labor equal employment 
opportunity agencies. 

ACK, the nation’s most powerful on-line career development resource, provides direct internet- 
based access for Americans needing job search assistance, career guidance, salary data and 
training and educational resources. ACK is a federal/state partnership financed through grants to 
states by the U.S. Department of Labor (DOL). Since 1997, ACK has received total DOL 
funding of $1 16.5 million, including $42.5 million for the current program year. 

ACK has four components that offer unique solutions to the increasing demands of today's labor 
market in meeting the specialized needs of job seekers, employers, workforce development 
professionals, and the training and education community. America’s Job Bank (AJB) is one of the 
four components, and is the most widely-used tool of ACK. Presently, AJB is an internet job 
search tool which allows both employers and job seekers to place job listings and resumes. 



RESULTS OF EVALUATION 



Our evaluation identified several concerns related to the development, implementation, and 
monitoring of ACK. More specifically, the issues identified were related to the AJB component 
of ACK. We did not identify any issues with the other components of ACK; therefore, we 
focused our evaluation on AJB. 

FINDING 1 - The Overall Effectiveness of America’s Job Bank is Difficult to Assess 

We identified several areas where ETA/ AJB can more effectively assess program performance. 
First, current measures do not capture all aspects of Job Bank performance. AJB's performance 
standard for PY 1999 is a one percent increase in the number of jobs posted on the AJB web site. 
Our analysis indicates that the number of job listings on the website is increasing. However, 
usage of the AJB web site is more difficult to assess; the number of server accesses ("web hits") is 
declining whereas the number of times the available job database is searched is showing modest 
improvement. These examples point to the need for multiple measures in order to fully assess 
performance. Using the performance standard “number of jobs posted” does not give a complete 
picture. We assert that program performance depends on a number of factors. In order to fully 
assess the performance of AJB, ETA should consider the effect of dimensions such as usage (web 
site utilization) and utility (to what degree did the web site aid individuals in finding employment). 
Second, current performance measures associated with AJB address program outputs rather than 
program outcomes. Current measures address the product (i.e., job listings) rather than the 
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impact the product has on customers (i.e., finding employment). The 1998 Workforce Investment 
Act (WIA) requires One-Stop Centers to collect data on program outcomes on a quarterly basis. 
This data includes information regarding the number of One-Stop users that have found jobs, how 
many are still employed, and how long they have held those jobs. While "self-service activities" 
(e.g., AJB) are exempt from this provision under Section 136(2)(A)(1), ETA should consider 
these measures in developing performance measures centered on program outcomes. 

Finally, WIA provisions emphasize customer service. AJB serves the needs of two types of 
customers, employers and job-seekers. ETA has conducted a study of employer satisfaction, but 
job seeker customer service has received significantly less attention. As job seekers are at the 
center of the purpose of AJB, we recommend that ETA devote significant effort to analyzing job 
seeker satisfaction. This analysis should be completed for registered as well as anonymous users. 

Given that AJB is an integral tool for providing American workers the access to employment 
assistance, there is a strong need for precise performance evaluation and program enhancement. 
Action on these items will enable AJB to streamline program delivery, more effectively serve its 
program recipients, and precisely gauge program performance. 

FINDING 2 - ETA Has Not Fully Addressed the Compliance Concerns of POL Agencies 
Responsible for Equal Employment Regulations 

The current version of AJB needs to further address the compliance needs of three equal 
employment opportunity agencies in the Department of Labor-the Office of Federal Contract 
Compliance Programs, the Civil Rights Center, and the Veterans’ Employment and Training 
Service. ETA is working with these agencies to resolve their concerns and has made some 
progress. However, our evaluation identified several outstanding areas pertaining to 
implementation of equal employment opportunity regulatory requirements that need to be 
addressed. These areas include: 

• Insufficient instructions to employers advising them of the regulatory requirement to 
collect demographic data and analyze the impact of their selection procedures. We 
recognize that responsibility for providing information on the regulatory requirements 
must come from the enforcing agency (i.e. OFCCP). Nevertheless, as the coordinator of 
the AJB in DOL, ETA must facilitate the process with OFCCP. 

• Current procedures to allow employers and EEO agencies to easily retrieve demographic 
information needed to comply with various record keeping requirements at various stages 
of applicant site searches are awkward and time consuming. 



• It is unclear if a higher response rate for filling out the electronic tear-off sheet would be 
obtained if it is placed either before or after the registration process; ETA and the equal 
employment agencies should revise the instructions for the tear-off sheet to encourage job 
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seekers to provide demographic information; the veteran sub-category of Campaign 
Veterans is currently not included on the self-identification section of AJB. 

• Limited English-proficient job seekers cannot effectively access AJB. 

• Written instructions on how to use the historical demographic database, including 
information on the database’s capabilities, are not readily available. 



RECOMMENDATION S 



We recommend the following actions be taken by ETA to further enhance AJB: 

1 . Conduct an on-line survey of America’s Job Bank web site users in order to determine the 
factors contributing to web site usage. 

2. Review Workforce Investment Act performance measures and other industry initiatives for 
the purpose of developing applicable outcome performance measures. 

3. Gather customer satisfaction data regarding America’s Career Kit/ America’s Job Bank 
customer service. 

4. Take a proactive approach to ensure that DOL regulatory requirements associated with data 
collection and impact analysis are available to employers using AJB. 

5. Coordinate with DOL enforcement agencies (OFCCP, CRC, VETS) to identify how they 
can assist employers in meeting their EEO and Affirmative Action record keeping 
responsibilities. 

6. ETA and EEO enforcement agencies together should: (a) determine where the electronic 
tear-off sheet should be inserted during the registration process in order to obtain the 
highest response rate, and (b) revise the instructions for the tear-off sheet to encourage job 
seekers to provide demographic information needed. 

7. Provide access to AJB for limited English-proficient job seekers, in accordance with 
Executive Order 13166. 

8. Incorporate the sub-category of Campaign Veterans to AJB self-identification section. 

9. Develop and disseminate written instructions on the use of the historical demographic 
database, including information on the potential capabilities of the database. 
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AGENCY COMMENTS AND OIG CONCLUSION 



In response to the official draft report, ETA provided suggested clarifications and modifications 
as well proposed corrective actions. Many of the suggested clarifications and modifications have 
been incorporated into this report. All others are addressed in the “Agency Response and OIG 
Conclusions” section of this report. ETA’s complete response is located in the Appendix section 
of this report. 



I BACKGROUND | 



The Employment and Training Administration (ETA) began using its discretionary funds in 
program year (PY) 1994 to support the development and implementation of America’s Labor 
Market Information System (ALMIS)-intemet based products and services. This system, 
combined with One Stop Career Center Grants, which were implemented in PY 1996, addressed a 
need of both job seekers and employers-current and quality information on employment 
opportunities and training programs at a locally centralized location. ALMIS is an integrated 
series of Labor market information systems linked to Federal, Regional, State and local 
constituents. America’s Career Kit (ACK) is a core component of ALMIS. 

ACK, the nation’s most powerful on-line career development resource, provides direct internet- 
based access for Americans needing job search assistance, career guidance, salary data and 
training and educational resources. ACK is a federal/state partnership financed through grants to 
states by the U.S. Department of Labor (DOL). ACK has four components that offer unique 
solutions to the increasing demands of today's labor market in meeting the specialized needs of job 
seekers, employers, workforce development professionals, and the training and education 
community. The following chart illustrates the inter-connectivity of the components of ACK. 



America’s Career Kit 




(1) America’s Job Bank (AJB), initially created in 1996, is the most widely-used tool in 
America’s Career Kit. Prior to 1996 there was a program in New York for distributing unfilled 
state jobs called the Interstate Job Bank, which eventually grew into AJB. Presently, AJB is an 
internet job search tool which allows both employers and job seekers to place job listings and 
resumes. As of February 2001, AJB had an average of 1.3 million jobs and 475,610 resumes on 
its website. 
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( 2 ) America’s Learning exchange (ALX) is a virtual yellow pages of training and education 
resources. It offers more than 300,000 courses and 6,000 training providers to the general public. 

(3) America's Career InfoNet (ACINet) is an information source for smart career decisions. An 
electronic storehouse of national, state, and metro labor market data not available elsewhere. 
Employment trends, wages and an analysis of prevailing salaries and living cost anywhere in the 
country. 

(4) America’s Service Locator (ASL) helps individuals to locate public service offices that assist 
in job-seeking, career planning, locating training, unemployment issues and employee-recruitment. 

The Occupational Network (0*NET) is the “common language” for these products, and appears 
on diagrams as the underlying coding scheme that allows the products to talk to each other, but is 
not a formal part of the ACK. 

The following chart shows ETA’s total allocated funding for ALMIS (S422 million) for PYs 
1997-2001, and the portion of those funds dedicated to ACK (SI 16.5 million) and AJB (S76 
million) 1 . 




Total Budget for ALMIS 
With Breakouts of ACK and AJB Allocations 
In $ Millions 
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Note: For Program Year 2001, $42.5 million for America’s Career Kit and $28 million for America’s 
Job Bank represent the budgeted funding levels; however these funds have not been allocated and invested in the 
products to date. 
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PURPOSE, SCOPE AND METHODOLOGY 



PURPOSE 



We conducted an evaluation of ETA’s development and implementation of ACK. Our evaluation 
covered Program Years (PY) 1997 through 2000-roughly the period from July 1, 1997 through 
June 30, 2001 . The evaluation was designed to assess whether ACK has: 

(1) developed adequate performance measures to gauge the overall success of the program, and 

(2) addressed the compliance requirements of DOL equal employment opportunity agencies. 



SCOPE 



During the course of our evaluation, we identified several concerns in the development, 
implementation, and monitoring of AJB. We did not identify similar problems with the other 
ACK components; therefore, we focused our evaluation on AJB. 



METHODOLOGY 

Our methodology employed the use of qualitative methods, quantitative methods and a document 
review as described below. 

Qualitative Methods 



We interviewed officials from the Civil Rights Center, ETA, Office of Federal Contract 
Compliance Programs, Office of the Solicitor of Labor and the Veterans’ Employment and 
Training Services. Representatives from these agencies offered their perspective on the strengths 
and weaknesses of AJB as they relate to equal employment opportunity compliance concerns. 

Quantitative Methods 

America’s Job Bank Service Center provided data regarding AJB web site usage and 
demographic characteristics of registered users. These data included: (1) frequency counts of 
server accesses ("web hits") over a 32-month span (from May 1998 to January 2001); 

(2) the number of job openings in each Occupational Information Network - Standard 
Occupational Classification (0*Net-SOC) category from May 1997 to January 2001; and 

(3) information regarding the required educational level of jobs posted on AJB and the number of 
job openings in each category of educational attainment. 
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Additionally, ETA provided information on ACK performance. These documents included 
analytical overviews of ACK components, enhancement plans, and two ETA-contracted 
evaluations of AJB. 

Document Review 



We examined documents regarding budget and program features, functionality, enhancements, 
and evaluations of ACK components. ETA provided us with analytical overviews of ALX, AJB, 
and America’s Career InfoNet, the ACK enhancement plan, ALX strategic plan, several monthly 
reports for both ALX and ACINet, and two ETA contracted evaluations of AJB. The first 
evaluation was an analysis of “e-mail feedback” provided by users of the AJB site, the second 
analyzed data collected from interviews with employer-users. Additionally, we received the ACK 
budgetary schedule focusing on PYs 1994-2000. 

We conducted our review in accordance with the Quality Standards for Inspections published by 
the President’s Council on Integrity and Efficiency. 
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FINDINGS AND RECOMMENDATIONS || 



FINDING 1 - Overall Effectiveness of America’s Job Bank is Difficult to Assess 



Our evaluation of AJB identified several areas where ETA/AJB can more effectively assess 
program performance. First, current measures do not capture all aspects of Job Bank 
performance. AJB's performance standard for PY 1999 is a one percent increase in the number of 
jobs posted on the AJB web site. We assert, however, that program performance depends on a 
number of factors. In order to fully assess the performance of AJB, ETA should consider the 
effect of dimensions such as usage (web site utilization) and utility (to what degree did the web 
site aid individuals in finding employment). 

Second, current performance measures associated with AJB address program outputs rather than 
program outcomes. Current measures address the product (i.e., job listings) rather than the 
impact the product has on customers (i.e., finding employment). The 1998 Workforce Investment 
Act (WIA) requires One-Stop Centers to collect data on program outcomes on a quarterly basis. 
This data includes information regarding the number of One-Stop users that have found jobs, how 
many are still employed, and how long they have held those jobs. While "self-service activities" 
(e.g., AJB) are exempt from this provision under Section 136(2)(A)(1), ETA should consider 
these measures in developing performance measures centered on program outcomes. 

Finally, WIA provisions emphasize customer service. AJB serves the needs of two types of 
customers, employers and job-seekers. While ETA has conducted a study of employer 
satisfaction, job seeker customer service has received substantially less attention. As job seekers 
are at the center of the purpose of AJB, we recommend that ETA devote significant effort to 
analyzing job seeker satisfaction. This analysis should be completed for registered as well as 
casual users (who account for the majority of site visits). 

Given that AJB is an integral tool for providing American workers the access to employment 
assistance, there is a strong need for precise performance evaluation and program enhancement. 
Action on these items will enable AJB to streamline program delivery, more effectively serve its 
program recipients, and precisely gauge program performance. 

(A) Current AJB Performance Measures Do Not Gauge All Aspects of Output Performance 

Data received from America’s Job Bank Service Center in New York permitted us to analyze 
ETA's current performance measure for AJB. According to the ETA Fiscal Year (FY) 1999- 
2004 strategic plan, the performance standard for AJB is a one percent increase in the number of 
jobs posted on the AJB web site. The benchmark for comparison utilized by ETA is the number 
of jobs posted on AJB in PY 1999. Figure 1 summarizes the trend in job postings. A large 
number of job listings are posted on the AJB web site, and this number is increasing. Based on 
this, we determined that the substantial number of job openings posted on AJB does facilitate 
employment opportunities for job seekers. 
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Further analysis indicates that the number of individuals visiting the AJB web site has increased 
substantially since the website’s launch, from roughly 19 million visits in May 1998, to 170 million 
visits in January 2001 (see Figure 2). Conversely, since August 1999 the number of server 
accesses ("web hits") has steadily declined. However, this measure does not fully capture site 
usage; ETA representatives assert that alternative measures (such as number of job searches) 
more effectively represent AJB usage. Efforts are currently under way to track this information; 
ETA representatives provided preliminary data regarding job searches. This data indicates some 
modest increases in the number of job searches. Given the limited time frame of this data, 
however, no credible conclusions can be made regarding the trend in job searches. Further, ETA 
officials acknowledge that growth in usage of the AJB web site has not kept pace with the number 
of available positions. The increasing gap between available jobs and site usage highlights the 
shortcoming of using a single performance measure. 

Figure 1. Figure 2. 

JOB LISTINGS AJB WEB SITE USAGE 
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Relying solely on one performance measure provides a skewed assessment of performance. As 
the charts above demonstrate, relying on a single measure could result in very different 
assessments regarding program performance. ETA's concerns about effective ways to measure 
site usage further reinforces the importance of multiple measures. Meeting goals regarding the 
number of jobs posted on AJB gives some insight into the program’s performance; however, all 
aspects of overall performance cannot be capture in this one measure. 

The slower rate of growth in site usage identified by ETA can be attributed to the popularity and 
growth of commercial sites. For example, Monster.com has experienced a compound quarterly 
growth rate of 20% since 1995 ( Monster.com press packet). With over 12 million current job 
seeker accounts, we can reasonably assume that some of the declines in usage of AJB can be 
attributed to gains by commercial sites. An evaluation commissioned by ETA makes similar 
conclusions. 




BEST COPY AVAILABLE 
14 





An evaluation, completed by Technical Assistance and Training Corporation (Washington, DC), 
focused on two primary weaknesses of the AJB web site. The first weakness is identified as 
“difficulty of use.” and the second as “customer service problems.” Combined, these two 
elements are the site’s most serious flaws. Despite high levels of market awareness, AJB has “low 
brand strength,” according to the Electronic Recruiting Index 2000 report. This means that 
individuals are aware of the self-directed employment services provided by AJB, but choose to 
utilize other sites. 

These concerns may be created or exacerbated by the “difficulty of use” of the AJB web site. 
Where commercial sites often require minimal information and “number of clicks” to retrieve job 
search information, the AJB’s information architecture is more cumbersome. The industry 
standard is currently three clicks. AJB's information architecture currently requires four or five 
clicks, depending on the specificity of the search and the options chosen. Consequently, workers 
may be electing to use commercial self-directed employment resources rather than those provided 
by ACK and specifically AJB. 

(B) Current Performance Measures Do Not Focus on Program Outcomes 

Based on observations noted in (A), AJB appears to provide access to a significant number of 
jobs; however, the lack of multiple performance measures impedes the ability to make solid 
conclusions regarding the overall effectiveness of AJB. These two items highlight the importance 
of focused performance measurement. According to the ETA FY 1997-2002 strategic plan, the 
performance goals related to ACK were: (1) increasing the number of employers listing jobs on 
AJB, and (2) increasing the number of resumes in ATB (America’s Talent Bank has since been 
combined into AJB). In the FY 1999-2004 ETA strategic plan, the relevant performance goals 
were changed to measure increases in the number of total job openings listed with AJB. ETA 
notes that these changes were included to reflect WIA requirements and could be altered further. 

The current ETA performance goal relevant to AJB measures the degree to which access to 
employment information is increasing. But this type of performance measure does not permit an 
assessment of how well the program is addressing its core purpose. Instead, an AJB performance 
measure based on increases in the number of currently available jobs focuses attention on program 
outputs rather than program outcomes. 

The ETA Five-Year Research Plan for FYs 2000-2004 echoes a similar theme. The Research 
Plan notes that the answers to four critical questions weigh heavily in determining the 
performance level of self-directed employment services. They are: 



1 . How can new technologies increase access to employment and training 
services for job seekers and employers? 

2. What are the demographic characteristics of individuals and employers that 
use AJB? 

3. To what extent do individuals obtain employment as a result of using AJB? 

4. Does AJB complement or substitute for local labor exchange services? 

Each of these questions deals with a particular aspect of performance measurement. Specifically, 
they prompt ETA to leam who is using the program, what outcomes the program produces, and 
how the program can be altered to improve performance. Current performance measures can 
answer none of these questions. 

ETA realizes the importance of outcome measurement. However, ETA has had difficulty in 
determining appropriate ways to measure outcomes. Commercial services such as Monster.com 
are not specifically interested in outcomes; they simply act as a job listing clearinghouse and 
derive their profits from advertising. Due to difference in motive, there is little industry consensus 
on outcome measurement. ETA has contracted a study of employment service industry trends in 
order to derive and shape appropriate outcome performance measures. 

One possible source of outcome measures is WLA. The implementation of WIA in 1998 
substantially altered the orientation of employment services from more intensive services to an 
emphasis on self-directed job search and career management through the One-Stop system. Job 
seekers visiting a One-Stop Center have access to core services provided by ACK through the 
internet. WIA requires that these One-Stop Centers submit quarterly reports on performance to 
appropriate state agencies; these data are compiled and forwarded to ETA. The performance 
measures for dislocated workers are: 

1 . The number of workers who have entered employment by the end of the first 
quarter after exit; 

2. (Of those employed in the first quarter after exit,) the number who are 
employed in the third quarter after exit; 

3 . (Of those employed in the first quarter after exit,) the earnings in the second 
and third quarter after exist; 

4. (Of those employed who received training services,) the number of workers 
who were employed in the first quarter after exit and received a credential 
(diploma, degree, other credential) by the end of the third quarter after exit. 
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These performance measures have been finalized and will take effect July 1, 2001, as reported in 
the May 3 1 , 200 1 , Notices section of the Federal Register. However, these performance 
measures are intended to measure the effectiveness of the One-Stop delivery system as a whole. 

In order to effectively gauge the performance of AJB, ETA should utilize WIA reporting 
requirements for the purpose of developing applicable outcome performance measures. 

These performance measures certainly address some aspects of self-directed employment services, 
as they will primarily be provided through the One-Stop System. While AJB is exempt from WIA 
reporting requirements under §136(2)(a)(l), we believe that these measures are a useful guide. 
Current performance goals relate only to the overall number of jobs available. In order to 
determine the effectiveness of ACK/AJB, performance measures must (as the ETA research plan 
notes) determine the demographic characteristics of individuals using AJB, and the extent to 
which individual job seekers obtain employment as a result of using AJB. 

(C) AJB Performance Measures Do Not Incorporate Customer Satisfaction Concerns 

A second feature of WIA that impacts performance measurement concerns customer satisfaction. 
WIA requires quarterly reporting of customer satisfaction data to ETA. The performance levels 
are measured by responses to a three-question battery that ask for job seeker reactions to the 
employment and job training services received at One-Stop Centers. No customer satisfaction 
measures exist with regard to AJB, even though previous evaluations noted that customer 
satisfaction/service is an important issue that requires attention. 

To satisfy the customer service measurement requirements of WIA, One-Stop Centers will utilize 
customer satisfaction surveys. These surveys, initially, will use the American Customer 
Satisfaction Index (ACSI). This index is currently used extensively in the business community, 
including thirty Fortune 500 companies, and in many European countries. The index is created 
through simple aggregation of scores from three specific questions targeting various aspects of 
customer service. These questions are: 

1. Overall, on a scale of 1 to 10, where “1” means “Very Dissatisfied” and “10” 
means “Very Satisfied,” how satisfied are you with the services? 

2. Considering all of the expectation you may have had about the services, to 
what extent have the services met your expectations? “1” now means “Met 
None of My Expectations” and “10” means “Met All of My Expectations.” 

3. Now I want you to think of the ideal program for people in your 
circumstances. How well do you think the services you received compare 
with the ideal set of services? “1” now means “Not Very Close to the Ideal.” 
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Surveys will be administered to both participants and employers, providing a wealth of 
information regarding program performance based on customer-reported perceptions. 

Previous evaluation of “customer satisfaction” with regard to AJB has been completed, but job 
seeker satisfaction has played a limited role. ETA has made progress toward addressing the 
concerns of job seekers through the implementation of a "pop-up" survey on the AJB web site. As 
a majority of the program's impact is derived from job seekers (i.e., individuals getting jobs), their 
perceptions regarding the successes and shortcomings of AJB are invaluable measures of 
performance. 

Conclusion 

Currently, ETA is making an effort to gauge AJB performance through program outputs. While 
ETA should continue to strive for more job listings on the AJB web site, ETA should increase its 
efforts to reliably measure the performance of AJB. Current performance measures do not 
account for different aspects of program performance. Reliance on a single measure (number of 
available jobs) hampers the future development of AJB and its ability to adapt its services to 
changing economic or workforce conditions. Additionally, the current performance measure 
addresses program outputs rather than program outcomes. Rather than an emphasis on whether 
AJB is helping people find and get jobs, the current measure emphasizes whether enough access 
to job information is available. 

Finally, ETA has conducted a study of employer satisfaction, but job seeker customer service has 
received significantly less attention. As job seekers are at the center of the purpose of AJB, we 
recommend that ETA devote significant effort to analyzing job seeker satisfaction. This analysis 
should be completed for registered as well as anonymous users. 



[Intentionally Left Blank] 



Recommendations 

We recommend that ETA: 

1. Conduct an on-line survey of America’s Job Bank web site users in order to determine the 
factors contributing to web site usage. Specifically: 

A. On visiting the AJB web site, include a “pop-up” window which informs visitors that 
a customer satisfaction survey is in progress, and should they be willing to participate, 
an email survey will be sent to them. Anonymity can be maintained through requiring 
only an e-mail address to send the survey. 

B. Design the survey in HTML format, or provide an HTML link in the email. This 
provides potential “interviewees” two opportunities to decline participation. The 
HTML format allows individuals to complete the survey and electronically “send” the 
information to appropriate data collection officials in a form readily convertible to 
spreadsheet or statistical software. 

C. Include questions which specifically target factors influencing site usage. For 
registered users, include a battery of questions that address reasons for registering on 
the site. Areas for assessment should minimally include the following questions: 

(1) How did the individual find out about AJB? 

(2) What are the individuars*de“mdgrapKic characteristics? t' — — — • — r — 

(3) How easy to use did the individual find the site? 

(4) How helpful was the site in finding appropriate job listings? 

(5) Would the individual use the AJB web site again? 

(6) Has the individual used commercial self-directed employment sites? 

(7) How does the AJB site compare to commercial sites? 

(8) Did the individual find a job based on information viewed on the AJB site? 

2. Review Workforce Investment Act performance measures and other industry initiatives 
for the purpose of developing applicable outcome performance measures. 

3. Gather customer satisfaction data regarding America’s Career Kit/America’s Job Bank 
customer service. 
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BEST COPY AVAILABLE 



FINDING 2 - ETA Has Not Fully Addressed the Compliance Concerns of DOL 
Agencies Responsible for Equal Employment Regulations 



The current version of AJB needs to further address the compliance needs of three equal 
employment opportunity agencies in the Department of Labor-the Office of Federal Contract 
Compliance Programs, the Civil Rights Center, and the Veterans’ Employment and Training 
Service. ETA is working with these agencies to resolve their concerns and has made some 
progress. However, our evaluation identified several areas pertaining to implementation of equal 
employment opportunity regulatory requirements. These areas include: 

• Insufficient instructions to employers advising them of the regulatory requirement to 
collect demographic data and analyze the impact of their selection procedures. We 
recognize that responsibility for providing information on the regulatory requirements 
must come from the enforcing agency (i.e. OFCCP). Nevertheless, as the coordinator of 
the AJB in DOL, ETA must facilitate the process with OFCCP. 

• Current procedures to allow employers and EEO agencies to easily retrieve demographic 
information needed to comply with various record keeping requirements at various stages 
of applicant site searches are awkward and time consuming. 

• It is unclear if a higher response rate for filling out the electronic tear-off sheet would be 
obtained if it is placed either before or after the registration process; ETA and the equal 
employment agencies should revise the instructions for the tear-off sheet to encourage job 
seekers to provide demographic information; the veteran sub-category of Campaign 
Veterans is currently not included on the self-identification section of AJB. 

• Limited English-proficient job seekers cannot effectively access AJB. 

• Written instructions on how to use the historical demographic database, including 
information on the database’s capabilities, are not readily available. 

Through interviews with representatives of ETA and the EEO agencies, and a review of agency 
regulations and other documents, we concluded that ETA did not initially include the EEO 
agencies in the design phase of AJB. We believe that early involvement of these agencies in the 
design phase would have eliminated many of the current problems, particularly those related to 
regulatory requirements. 

Below we summarize each of the agency’s authorizing regulations, identify the current problem 
areas we identified that are associated with AJB, and make recommendations for resolution. 
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(A) The Office of Federal Contract Compliance Programs, Employment Standards 
Administration 



The Office of Federal Contract Compliance Programs (OFCCP) has the responsibility of assuring 
that employers doing business with the Federal Government comply with the equal employment 
opportunity and affirmative action provisions of their contracts. OFCCP administers and enforces 
three EEO programs: Executive Order 1 1246; Section 503 of the Rehabilitation Act of 1973; and 
the affirmative action provisions of the Vietnam Era Veterans’ Readjustment Assistance Act of 
1974. 

OFCCP’s regulations require three types of record keeping regarding federal contractors’ 
employment processes, described below: development and maintenance of information to monitor 
employee selection processes (41 CFR Part 60-3); general record retention (41 CFR 60-1. 12(a)); 
and development and maintenance of records and analyses necessary to an affirmative action 
program (41 CFR 60-1.40 and 60-2.1). If this information is not maintained by federal 
contractors who are using the AJB as one of its recruitment sources, and such a contractor is 
selected by OFCCP for a compliance evaluation, the agency is not able to conduct its required 
analyses of selection and hiring procedures. We recognize that it is the employer’s responsibility 
to collect such data, but ETA should provide a system for information retrieval that is not 
cumbersome or time consuming. In other words, it should allow for easy retrieval of the data. 

(1) General Data Requirements under the Uniform Guidelines on Employee Selection 
Procedures 

Section 41 CFR 60-3.4(A) requires that each contractor “...maintain and have available 
for inspection records or other information which will disclose the impact which 
its... selection procedures have upon the employment opportunities of persons by 
identifiable race, sex, or ethnic groups...” 2 

Contractors with 100 or more employees 3 are more specifically required by 41 CFR 60- 
3.15A(2) to maintain and have available records for each job on applicants, hires, 
promotions and terminations (as well as any other selection decisions) by sex and by 
each minority group for whom EEO-1 reporting is required. These records must be 
sufficient to disclose the impact of the selection process on women and on each minority 
group for each job. Contractors are then required to conduct adverse impact analyses 
for women and for each minority group that constitutes 2% or more of the relevant 



2 Such identifiable groups are defined by 41 CFR 60-3.4(B) as those groups for whom EEO-1 reporting is 
required-i.e., African Americans, Hispanics, Asian/Pacific Islanders, American Indians/ Alaskan Natives-along 
with Whites and totals. 

Contractors with fewer than 1 00 employees must maintain the records identified in 4 1 CFR 60- 
3.15A(l)-e.g. the number of applicants and persons hired, promoted and terminated by sex and by each minority 
group which constitutes 2% or more of the labor force in the relevant labor area. Such small employers are not 
required to conduct adverse impact analyses of these data. 
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labor area or of the applicable internal workforce. Where a contractor determines that a 
selection process has an adverse impact, evidence of validity, as described in 4 1 CFR 
Part 60-3 must also be maintained and made available for review during a compliance 
evaluation by OFCCP. 

(2) General Record Retention Requirements 

The general record retention regulations at Section 60-1. 12(a) are applicable to all 
contractors covered by Executive Order 1 1246, as amended (except those exempted 
under 41 CFR 60-1.5). As such, contractors are required to retain any personnel or 
employment record made or kept by the contractor, including records pertaining to 
hiring, applications and resumes, tests and test results, and interview notes. Contractors 
with at least 150 employees and a Government contract of at least $150,000 must keep 
such records for not less than two years from the date a record is made or the date of 
the personnel action it concerns, whichever occurs later. Contractors that employ fewer 
than 150 employees and/or that do not have a Government contract of at least $150,000 
must keep such records for a period of one year. 

(3) Affirmative Action Record Keeping Requirements 

Those contractors required to develop a written affirmative action program (i.e. those 
with at least 50 employees and at least one contract valued at $50,000), must also 
collect and maintain information related to recruitment and hiring processes. They must 
also analyze their hiring practices for the past year, including recruitment sources (41 
CFR 60-2. 17(b), (c), (d)). Additionally, they must compile and maintain support data 
for the analyses performed in the development of the Affirmative Action Program. 
Applicant flow 4 data should include applicants identified through the Internet, as well as 
more traditional recruitment sources. 

Compliance Requirements Related to AJB 

Based on these requirements, we concluded there are three major outstanding compliance 
requirement areas associated with OFCCP enforcement that have not been resolved in the current 
version of AJB. 

(1) The AJB should include instructions to employers informing them of their record 

keeping obligations, and the need to conduct adverse impact analyses when applicable. 



4 Applicant flow log is defined in Chapter 1 of the Federal Contract Compliance Manual as a 
“chronological compilation of applicants for employment or promotion, showing the persons categorized by race, 
sex, and ethnic group, who applied for each job title ...during a specific period.” 
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(2) We found that federal contractors who are using the AJB must go through a series of 
steps in order to collect the demographic information required for each stage of their 
applicant search. The process for retrieving the information is not user friendly. First 
the employer must send a written letter to the AJB service center in New York, 
specifying its search criteria and requesting the pertinent data for each search it has 
conducted, then the service center normally takes several weeks to send a written reply 
to the employer. 

With over 100,000 supply and service contractor establishments who may fill dozens, or 
even hundreds, of job vacancies each year, this system clearly is not efficient and will 
become an even greater problem in future years. Also, we note that in addition to the 
Federal contractors for whom OFCCP has responsibility, there are millions of other 
business that are not Federal contractors but are subject to the Uniform Guidelines on 
Employee Selection Procedures. These business are also required to maintain the 
required record keeping information. 

We believe that this retrieval system is also inadequate to meet the compliance needs of 
OFCCP when conducting an evaluation. They would be faced with the same challenges 
as the employer in attempting to retrieve the required data in order to verify adverse 
impact in the selection process. 

(3) The electronic tear-off sheet that includes the request for the demographic information 
of race, sex and disability status is at the conclusion of the resume process, and does not 
encourage users to provide pertinent information. The request for information on 
veteran status is a mandatory field at the beginning of the job seeker’s resume posting 
process. All of the demographic information should be presented together and job 
seekers should be encouraged to provide the information. 

(B) Office of the Assistant Secretary of Administration and Management 
Civil Rights Center 

The Civil Rights Center (CRC) is a component of the Office of the Assistant Secretary of 
Administration and Management. CRC enforces several Federal statutes and regulations that: (1) 
prohibit discrimination in DOL funded programs and activities and (2) prohibit discrimination on 
the basis of disability by certain public entities and in DOL assisted activities. CRC enforces DOL 
policies, practices, and procedures under Title VI of the Civil Rights Act of 1964, 

Title II of Americans with Disabilities Act of 1990, Section 504 of the Rehabilitation Act of 1973, 
Section 508 of the Rehabilitation Act of 1973, Age Discrimination Act of 1975, Title IX of the 
Education Amendments of 1972, Section 188 of the Workforce Investment Act of 1998, 
Executive Order 13160, and Executive Order 13166. 



15 



ERIC 



23 



CRC is involved in the continuing development and implementation of AJB to ensure that 
neither AJB nor its users (e.g., United States Employment Service), discriminate on a prohibited 
basis, and that all job seekers have equal access to AJB. 

Compliance Requirements Related to AJB 

CRC identified several areas of concern similar to those listed above for OFCCP - such as record 
keeping needs of all employers, demographic information on applicants. In addition to those 
issues we note one additional outstanding concern in AJB that falls under the purview of the 
CRC-the lack of access that non-English speaking applicants have to the system. These 
applicants are referred to as Limited English Proficient. 

In 1998, CRC urged ETA to accommodate potential AJB users who have Limited English 
Proficiency (LEP), based on the requirements of Title VI of the Civil Rights Act of 1964. Also, 
Executive Order 13166 (EO 13166), signed in August 2000, further emphasized the importance 
of this accommodation. 

Anything a federal agency does, including all contact with the public, falls within the scope of the 
term federally assisted programs or activities. The definition of federally assisted programs or 
activities used under EO 13166 is consistent with the definition provided under the regulations for 
application of Section 504 of the Rehabilitation Act of 1973. EO 13166 requires all federal 
agencies take reasonable steps to provide meaningful access to their own federally assisted 
programs or activities. 

ETA has made attempts to integrate LEP standards in ACK/AJB since 1998. ETA, in 
conjunction with the Interstate Conference of Employment Security Agencies/National 
Association of State Workforce Agencies, identified problems that existed in providing services to 
LEP individuals at both the national and state levels through the Public Electronic Labor 
Exchange initiative, known as PELEX. Several options were discussed in meetings between CRC 
and ETA. ETA made a substantial effort to address this issue through the implementation of the 
Babblefish language translation software-a translation service that removes language barriers 
across the World Wide Web, which was added to previous versions of the National AJB site. 
Unfortunately, ETA determined that Babblefish was problematic because it could not translate 
“dynamic” web pages (e.g., if an LEP job-seeker was to enter a resume into the system in a 
language other than English, access to the resume would be limited. The resume would only be 
translated and accessed if an employer was searching the website in the language that the resume 
was entered). 

As of the date of this report, ETA has reactivated Babblefish until a system called Websphere is 
implemented in the 2002 AJB update. Both ETA and CRC anticipate that implementation of 
Websphere will resolve the LEP issue. 
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(C) The Veterans’ Employment and Training Service 



The Veterans’ Employment and Training Service (VETS) is charged with the responsibility to 
assist veterans, reservists, and National Guard members in securing employment, including the 
rights and benefits associated with employment, through existing programs, the coordination and 
merger of programs, and the implementation of new programs. In addition to other 
responsibilities, the VETS resolves claims by veterans, reservists, and National Guard members 
under the Uniformed Services Employment and Reemployment Rights Act, and investigates 
alleged veteran preference violations. Additionally, they provide employment and training 
services to eligible veterans through grants to States, local governments, and non-profit agencies. 
VETS are mandated to provide services as outlined in Title 38 of the United States Code (USC) 
Chapter 41 sections 4101, 4102 and 4212, as well as, 20 CFR Chapter IX Part 1000-1009. 



Compliance Requirements Related to AJB 

(1) Demographic information on AJB does allow job seekers to self-identify as a veteran. 
However, VETS, in accordance with 5 USC Sec. 2108, requires that job seekers be 
allowed to further self-identify as Campaign Badge Veterans. This sub-category has not 
been included on AJB. In August 2001, ETA provided documentation that outlined its 
plan to add this sub-category to its next AJB version, scheduled for September 200 1 . 

(2) ETA has completed the process of creating a demographic historical database and has 
provided a physical demonstration of the database to VETS. However, no written 
instructions have been provided to potential users on how to utilize the database. 



(Intentionally Left Blank] 
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Recommendations 



We recommend that ETA: 

4. ETA, with EEO enforcement agencies, must take a proactive approach to ensure that 
DOL regulatory requirements associated with data collection and impact analysis are 
available to employers using AJB. 

5. ETA should coordinate with DOL enforcement agencies (OFCCP, CRC, VETS) to 
identify how they can assist employers in meeting their EEO and Affirmative Action 
record keeping responsibilities. 

6. ETA and EEO enforcement agencies together should: (a) determine where the electronic 
tear-off sheet should be inserted during the registration process in order to obtain the 
highest response rate, and (b) revise the instructions for the tear-off sheet to encourage 
job seekers to provide demographic information needed. 

7. Provide access to AJB for limited English-proficient job seekers, in accordance with 
Executive Order 13166. 

8. Incorporate the sub-category of Campaign Veterans to AJB self-identification section. 

9. Develop and disseminate written instructions on the use of the-historical demographic _ „ 
database, including information on the potential capabilities of the database. 



1 AGENCY RESPONSE AND OIG CONCLUSIONS || 

ETA’s Response to Executive Summary 

• Finding 1 - The Overall Effectiveness of America's Job Bank is Difficult to Assess 

"We still believe that this should be reworded. This issue is not confined to America's Job Bank , 
but is a generic problem inherent in all web-based, self-service systems. This title makes it 
sound as though other such systems have addressed and resolved these issues, while AJB has 
done nothing. This is not the case. While it is true that the current performance measures used 
in conjunction with AJB don 't focus on outcome measures, we have met or exceeded the 
measures that have been used. The substantive issue that needs to be addressed is that there are 
no generally accepted performance measures for web-based self-service systems, and those used 
for staff-assisted do not fit and cannot be used. To be fair, AJB has been a leader in trying to 
address this issue . " 

OIG Conclusion for Executive Summary 

We believe our conclusion that the overall effectiveness of America's Job Bank is difficult to 
assess is accurate. This conclusion is based on evidence that usage is difficult to determine, 
existing performance measures do not adequately address web site utility, and there are concerns 
regarding customer service issues. We do not believe that one can infer from the title that other 
systems have addressed and resolved these issues while AJB has done nothing. Based on this, we 
are leaving the title of Finding One as written. 

FINDING ONE RECOMMENDATIONS 

RECOMMENDATION ONE 

ETA’s Response Recommendation One; 

• Recommendation 1: Conduct an on-line survey of America’s Job Bank web site 

users in order to determine the factors contributing to web site 
usage. Specifically: 

A. On visiting the AJB web site, include a “pop-up” window which informs visitors 
that a customer satisfaction survey is in progress, and should they be willing to 
participate, an email survey will be sent to them. Anonymity can be maintained 
through requiring only an e-mail address to send the survey. 

B. Design the survey in HTML format, or provide an HTML link in the email. 

This provides potential “interviewees” two opportunities to decline 
participation. The HTML format allows individuals to complete the survey and 
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electronically “send” the information to appropriate data collection officials in a 
form readily convertible to spreadsheet or statistical software. 



C. Include questions which specifically target factors influencing site usage. For 
registered users, include a battery of questions that address reasons for 
registering on the site. Areas for assessment should minimally include the 
following questions: 



( 1 ) 

( 2 ) 

(3) 

(4) 

(5) 

( 6 ) 

(7) 

( 8 ) 



How did the individual find out about AJB? 

What are the individual’s demographic characteristics? 

How easy to use did the individual find the site? 

How helpful was the site in finding appropriate job listings? 

Would the individual use the AJB web site again? 

Has the individual used commercial self-directed employment sites? 

How does the AJB site compare to commercial sites? 

Did the individual find a job based on information viewed on the AJB site? 



"We do not agree with this recommendation. Doing an on-line survey is very expensive and 
time-consuming and feel that there are better and less costly ways to gain this same type of 
information, namely focus groups, e-mail feedback and other means of user input. Again, it 
makes little sense to spend more money to determine a customer's satisfaction with a service, 
than it does to provide the service itself. " 

OIG Conclusion 



We consider this matter unresolved. In order to attain the goal of "becoming a market leader in 
self-directed employment services," ETA must be able to determine: (1) how useful individuals 
find the material on the AJB web site; and (2) how satisfied individuals were with various service 
aspects (e.g., support, troubleshooting). Site utility and customer service should therefore be 
paramount concerns. ETA should make concerted efforts to further study these items. While we 
believe that a web survey is the best option considering the costs and benefits, we acknowledge 
that other avenues for obtaining this information are equally valid. To this end, ETA should 
provide a plan detailing their proposed actions. The plan should include: (1) details of the 
approach to be used (e.g., phone survey, mail survey, focus group, sample size, etc); (2) strategies 
for addressing concerns listed in Recommendation 1 :C(1)-C(8); (3) a copy of the data collection 
instrument (survey or interview questions, interviewer protocol); and (4) a project timeline. This 
plan should be submitted to this office no later than December 1 7, 200 1 . 
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RECOMMENDATION TWO 



ETA’s Response to Recommendation Two 

• Recommendation 2: Review Workforce Investment Act performance measures and 

other industry initiatives for the purpose of developing 
applicable outcome performance measures. 

"As pointed out in our previous comments to your office, self-service systems are exempted from 
the core WIA performance measures. However, we agree with your recommendation as it has 
been rewritten and have already begun work on this recommendation. " 

OIG Conclusion 

We consider this recommendation to be resolved. It will be closed upon receipt of a status report 
to this office by December 17, 2001, indicating actions taken to date and planned future actions. 

RECOMMENDATION THREE 

ETA’s Response to Recommendation Three 

• Recommendation 3: Gather customer satisfaction data regarding America’s Career 

Kit/America’s Job Bank customer service. 

"We agree with this recommendation, although it lacks specificity. We need to begin work on 
determining what type of customer satisfaction data is relevant to the electronic tools. " 

OIG Conclusion 



We consider this recommendation to be resolved. It will be closed upon receipt of a status report 
to this office by December 17, 2001, indicating specific determinations regarding relevant 
customer service data, actions taken to obtain this data, and planned future actions. 

FINDING TWO RECOMMEND A TIONS 

RECOMMENDATION FOUR 

ETA’s Response to Recommendation Four 

• Recommendation 4: ETA, with EEO enforcement agencies, must take a proactive 

approach to ensure that DOL regulatory requirements 
associated with data collection and impact analysis are 
available to employers using AJB. 
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“It is not appropriate for ETA or AJB to advise employers of an OFCCP regulatory 
requirement. Per the February 22, 2000 agreement, we agreed to provide a link to an OFCCP 
web page and label it '... to find out more about your responsibilities as a Federal Contractor, 
click here. ’ That link exist today in the employers ’ registration process, and comes up whenever 
a registering employer self-identifies as a federal contractor. We believe this effectively 
addresses this issues. We believe we have met the spirit of this finding and it should be 
removed". 

OIG Conclusion 



We consider this recommendation unresolved. ETA should notify OFCCP in writing that OFCCP 
is to ensure that information provided in their link is current and adequately addresses record- 
keeping responsibilities of Federal Contractors. In order to close the recommendation, ETA 
should submit to this office a copy of their correspondence to OFCCP and the response received 
from them by December 17, 2001, detailing the proposed actions to be taken. 

RECOMMENDATION FIVE 

ETA’s Response Recommendation Five 

• Recommendation 2: ETA should coordinate with DOL enforcement agencies 

(OFCCP, CRC, VETS) to identify how they can assist 
employers in meeting their EEO and Affirmative Action record 
keeping responsibilities. 

“We agree with this recommendation, and have launched an effort to identify how we can assist 
employers to meet their EEO and Affirmative Action record keeping responsibilities called the 
EEO/AA Tools Workgroup. Participating in this effort are representatives from OFCCP, Equal 
Employment Opportunity Commission, Civil Rights Center, and VETS. ” 

OIG Conclusion 



We consider this recommendation to be resolved. It will be closed upon receipt of a status report 
to this office by December 17, 2001, indicating actions taken to date and planned future actions. 

RECOMMENDATION SIX 

ETA’s Response Recommendation Six 

• Recommendation 6: ETA and EEO enforcement agencies together should: 

(a) determine where the electronic tear-off sheet should be inserted during the 
registration process in order to obtain the highest response rate, and (b) revise the 
instructions for the tear-off sheet to encourage job seekers to provide demographic 
information needed. 



22 




30 



“We don ’t believe that moving the tear-off sheet to the beginning of the process will increase the 
response rate, but agree to look in to what location would provide the best response rate. 
However, there is a cost associated with moving this information in AJB, and we would object to 
moving it back to the end of the process if the move to the front of the process netted poorer 
results. In addition, the language to “encourage ” job seekers to provide the demographic 
information was provided by the enforcement agencies requesting the information, we are not 
opposed to changing the language if they provide us with language they think would better suit 
this purpose. ” 

OIG Conclusion 



We consider this recommendation to be resolved. ETA, in conjunction with EEO enforcement 
agencies, should determine the most effective placement for the electronic tear off sheet. ETA 
should also request in writing that appropriate EEO enforcement agencies review existing 
language to encourage job searches to provide demographic information and provide an update to 
ETA if necessary. Please submit a status report outlining the actions taken by ETA by 
December 17, 2001. 

RECOMMENDATION SEVEN 

ETA’s Response Recommendation Seven 

• Recommendation 7: Provide access to AJB for limited English-proficient job 

seekers, in accordance with Executive Order 13166. 

“ It should be noted that the free online translator operated by Alta Vista known as Babel Fish, 
which was previously on the AJB website, has been reactivated and is currently available for 
translation services. It was taken down when we decided to implement a better solution to the 
problem using our new enterprise software form IBM called Websphere which includes intrinsic 
support for multiple languages. However, it is taking longer to implement Websphere than 
originally planned, so that Alta Vista option has been reactivated and will remain on the web site 
until Websphere can be fully implemented. “ 

OIG Conclusion 



We consider this recommendation to be resolved. It will be closed when ETA provides 
information to this office on IBM’s Websphere enterprise software. The information should be 
provided no later than July 16, 2002. 
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RECOMMENDATION EIGHT 



ETA’s Response to Recommendation Eight 

• Recommendation 8: Incorporate the sub-category of Campaign Veterans to AJB 

self-identification section. 

“We agree with this recommendation. Per our agreement with the VETS, the Campaign 
Veterans category will be added in the September release. However, there is the following point 
of clarification; the veterans ’ information is not currently part of the “electronic tear-off sheet ” 
and will not be in the September release. It is kept separate format the other information, as we 
are required by the law to ask about veteran status and collect this information, but we do not 
have th authority to require the collection of demographic information. ” 

OIG Conclusion 



We consider this recommendation resolved. It will be closed when the update is complete and 
such action is reported to this office no later than December 17, 2001. 

RECOMMENDATION NINE 

ETA’s Response to Recommendation Nine 

• Recommendation 9: Develop and disseminate written instructions on the use of the 

historical demographic database, including information on the 
potential capabilities of the database. 

“We are assuming that since you removed ‘interested parties 'from this recommendation per our 
last round of comments that you are referring to the enforcement agencies. If this is the case we 
believe that we have already provided this information to the enforcement agencies, but we can 
repeat this demonstration, in necessary, as well as provide a written memo documenting the 
procedures. If this assumption is incorrect, and a broader audience is intended, we don ’t agree 
with this recommendation. ” 

OIG Conclusion 



We concur with ETA’s proposed action and consider the recommendation to be resolved. It will 
be closed when ETA provides information regarding the demonstration presentation and written 
procedures. Please provide the information no later than December 17, 2001. 
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APPENDIX 

COMPLETE AGENCY RESPONSE 
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U.S. Department of Labor 



Assistant Secretary for 
Employment and Training 
Washington, D.C. 20210 



SEP | 7 200! 



MEMORANDUM FOR: WILLIAM H. PICKLE 



Assistant Inspector General 

Office of Communications, Inspections and Evaluations 



FROM: 



Assistant Secretary 
Employment and Trainin 



EMILY STOVER DERC 




SUBJECT: 



Comments in Response to the Official Draft Report on the 
Evaluation of America’s Career Kit (No. 2E-03-39O-OOO2) 



Attached are our written comments in response to the official draft report of your evaluation of 
the Employment and Training Administration’s (ETA) development and implementation of 
America’s Career Kit. 

While you considered and incorporated many of our comments to the preliminary draft report 
into this official draft, we have attached some additional comments for your consideration and 
inclusion in the final report. 

I again appreciate the opportunity to provide you with these comments and hope you will take 
them into consideration in your final report. If you have any questions, please do not hesitate to 
contact David Moiman of my staff at (202) 693-3691 . 



Attachment 




A Proud Member of America’s Workforce Network 



Comments on Official Draft Report on the 
Evaluation of America's Career Kit (No. 2E-03-3 90-0002) 



Executive Summary- page iii. Finding 1 : “The Overall Effectiveness of America s Job 
Bank is Difficult to Assess.” We still believe that this should be reworded. This issue is 
not confined to America’s Job Bank, but is a generic problem inherent in all web-based, 
self-service systems. The title makes it sound as though other such systems have 
addressed and resolved these issues, while AJB has done nothing. This is not the case. 
While it is true that the current performance measures used in conjunction with AJB 
don't focus on outcome measures, we have met or exceeded the measures that have been 
used. The substantive issue that needs to be addressed is that there are no generally 
accepted performance measures for web-based self-service systems, and those used for 
staff-assisted do not fit and cannot be used. To be fair, AJB has been a leader in trying to 
address this issue. 

Rxecutive Summary, page iii. first paragraph under Finding ll “However, since August 
1 999, the number of server accesses (“web hits”) has steadily decreased.” “Web hits” are 
a notoriously poor measure for gauging performance. Hits represent the number of server 
accesses in loading a web page. Depending on how the page is constructed, this number 
t» gn vary tremendously over tim e as new versions and more efficient code is developed 
and implemented. We previously pointed out two blips in the graph representing just 
such occurrences in AJB and subsequently provided the OIG additional statistical 
information about AJB performance over time, showing the number of user sessions and 
the number of job and resume searches, which we believe are much better indicators of 
the usage of the AJB web site. Both of these indicators showed growth over the previous 

year. 

FyftMfiive Summary, nane iv. second paragraph; “ETA has conducted a study of 
employer satisfaction, but job seeker customer service has received significantly less 
attention.” This is incorrect. Focus groups and analysis of e-mail feedback was 
perforated for both job seekers and employers, not just employers. We believe that the 
review team came to this conclusion because of the studies that we provided to them. We 
hnit actually paid for some research on the employer customer, because we believed that 
the AJBSC had over focused on feedback from the job seeker customer. We suggest the 
OIG interview staff at the AJB Service Center to determine the steps that they routinely 
take to identify customer satisfaction and user feedback, particularly from job seeker 
customers. 

Rxacutive Summary, pace iv. first bull et under Finding 2j “Nevertheless, as the 
coordinator of the AJB in DOL, ETA must facilitate the process with OFCCP.” It has 
always been ETA’s position that the responsibility for advising employers how to comply 
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with an OFCCP regulatory requirement was OFCCP’s responsibility. Per the February 
22, 2000 agreement, we agreed to provide a link to an OFCCP web page and label it ... 
to find out more about your responsibilities as a Federal Contractor, click here.” That 
link exists today in the employers’ registration process, and comes up whenever a 
registering employer self-identifies as a federal contractor. We don t know how we could 
facilitate this process any further. 

F.v^cntive Summary, page iv. second bullet und er Finding 2: There are procedures in 
place to allow employers and EEO agencies a way to retrieve the demographic 
information on searches they conducted in AJB. A letter is sent to the AJB Service 
Center stating the date and time of their search and the search criteria used. The search is 
re-created on the historical database, and the information they need on applicant profiles 
is retrieved and forwarded to them. To ensure the security of this very private 
information, the database that contains characteristic and demographic data must be kept 
off-line. We can work with the enforcement agencies to develop a better process, but 
they must realize that because this sensitive data has to be kept off-line, it will never be 
able to be retrieved at the point of the actual search (i.e., real-time). 

punitive Summary, nage v. fourth bullet under Finding 2 ; “Limited English-proficient 
job seekers cannot effectively access AJB.” It should be noted that the free online 
translator operated by Alta Vista known as Babel Fish, which was previously on the AJB 
web site, has been reactivated and is currently available for translation services. It was 
taken down when we decided to implement a better solution to the problem using our 
new enterprise software from IBM called Websphere which includes intrinsic support for 
multiple languages. However, it is taking longer to implement Websphere than originally 
planned, so the Alta Vista option has been reactivated and will remain on the web site 
until Websphere can be fully implemented. 

pYftr.ntive Summary, page v. Recommendation s, first bullet: We do not agree with this 
recommendation, as it is very vague in its purpose of determining “factors contributing to 
web site usage.” Doing an on-line survey is very expensive and time-consuming and we 
feel that there are better and less costly ways to gain this type of information, namely 
focus groups and other means of user input. 

F.vfirntive Summary, page v. Recommend ations, second bullet: As pointed out in our 
previous comments to your office, self-service systems are exempted from the core WIA 
performance measures. However, we agree with your recommendation as it has been 

rewritten and have already begun work on this recommendation. One of the NET 

Board’s three strategic goals is for the ACK web sites to be “market leaders. In this 
regard, we are working to identify the metrics that can be used to define the term market 
leader” We will then benchmark the marketplace using those metrics and applying the 
results to our web sites. While these measures may not equate to WIA outcomes, they 
should provide a sense of the value of the web site to the public. Further, ETA is also 
already funding a survey designed to estimate employment outcomes associated with 

using AJB. 
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9 Fvpriitive Summary, pace v. Recommend ations, third bullet: We agree with this 

recommendation, although it lacks specificity. We need to begin work on determining 
what type of customer satisfaction data is relevant to the electronic tools. 

1 0. F.-xe^ptive Summary, page v. Recommen dations, fourth bullet: We believe we have 

already met the spirit of this recommendation, as discussed in comment number 4 . We 
believe this recommendation should be removed. 

1 1 F.vRcutive Summary, nage v. Recomme ndations, fifth bullet: We agree with this 
recommendation, and have already launched an effort to identify how we can assist 
employers to meet their EEO and Affirmative Action record keeping responsibilities 
called the EEO/AA Tools Workgroup. Participating in this effort are representatives 
from OFCCP, EEOC, Civil Rights Center, and VETS. 

12 Fvpmtive Summary, page v. Recommen dations, sixth bullet: We don t believe that 
moving the tear-off sheet to the beginning of the process will increase the response rate, 
but agree to look into what location would provide the best response rate. However, there 
is a cost associated with moving this information in AJB, and we would object to moving 
it back to the end of the process if the move to the front of the process netted poorer 
results. In addition, the language to “encourage” job seekers to provide the demographic 
information was provided by the enforcement agencies requesting the information; we are 
not opposed to changing the language if they provide us with language they think would 
better suit this purpose. 

1 3 Fyenitive Summary, page v. Recommen dations, seventh bullet: Agree. Per comment 
number 6, Babel Fish is back on the site and will remain on it until such time as the 
Websphere technology can be fully implemented. 

1 a Fxpciitive Su mmary, nage v. Recommendation s, eighth bullet; Agree. Per our agreement 
with the VETS, the Campaign Veterans category will be added in the September release. 
However, there is the following point of clarification; The veterans’ information is not 
currently part of the “electronic tear-off sheet” and will not be in the September release. 

It is kept separate from the other information, as we are required by law to ask about 
veteran status and collect this information, but we do not have the authority to require the 
collection of demographic information. 

1 5 Executive Summary, nage v. Recommendations , ninth bullet; We are assuming that since 
you removed “interested parties” from this recommendation per our last round of 
comments that you are referring to the enforcement agencies. If this is the case, we 
believe that we have already provided this information to the enforcement agencies, but 
we can repeat this demonstration, if necessary, as well as provide a written memo 
documenting the procedures. If this assumption is incorrect, and a broader audience is 
intended, we don’t agree with this recommendation. 
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Rarlf prmind. pace 2. f2) America’s Learning exchange and (3) America’s Career 
TnfoNet: The descriptions for these two remained the same from the previous draft and 
are still not wholly accurate or are incomplete. A fact sheet describing each component is 
attached again, for your information. 

Rarkpround. page 2, , Bar Chart: While some corrections were made to this chart, it still 
requires some additional clarification/edits. It should be noted that the $1 .5 million for 
AJB in PY 1997 came from the ES budget, not the ALMIS budget. In addition, no PY 
2001 funds have been released to AJB yet. Although a release of funds to AJB is 
currently in process, only $22 million will be released, not the entire $28 million 
budgeted as currently depicted on the chart. In its current form, the chart overstates what 
has been allocated to AJB. This can be corrected in one of two ways. Either the PY 2001 
data could be dropped from the chart entirely, which may be appropriate, since the 
evaluation only covers PY 1997 through PY 2000, according to the first paragraph of die 
Executive Summary. Or a footnote can be added to the chart that $42.5 million and $28 
million for the ACK and AJB respectively is the budgeted funding levels, but have not 
been allocated and invested in the products to date. 

Purpose- Scope and Methodology, page 3. M ethodology. Qualitative Methods: While it 
is now clear that staff from the America’s Job Bank Service Cotter were not interviewed, 
we believe that they should have been. AJB is operated by the Service Center through a 
grant to the State of New York, it is not a “federal labor market information system.” 

Purpose. Scope and Methodology, pace 3. Methodology. Quantitative Methods: The 
corrected OIG report refers to “(1) frequency counts of server accesses (“web hits”) over 
a 32-month span” as data provided by die Service Center. It doesn’t refer to the 
additio nal, more accurate information (the number of AJB user sessions in the same 
quarter in PY1999 and in PY2000 and the number of job and resume searches that were 
conducted in PY1999 and in PY2000) that were also provided. Server accesses are not a 
good or consistent measure for gauging site usage, as navigation improvements in new 
versions of the site tend to reduce server accesses dramatically. We hope that you will 
use this additional, more accurate and more definitive information in drawing your 
conclusions. 

Purpose: Scone and Methodology, page 4. Methodolog y. Document Review; In the 
second sentence America’s Career InfoNet is improperly referred to as “America’s Career 
0*Net." 

Findings and Recommendations. Finding h To more accurately reflect the situation, we 
recommend rewording this finding, as per comment number 1 . 



Findings and Recommendations, page 5. paragraph 3: Just as in comment number 3, the 
statement “job seeker customer service has received significantly less attention” is 
incorrect As stated before, focus groups and analysis of e-mail feedback was performed 
for both job seekers and employers, not just employers. Once again, we suggest that had 



the OIG interviewed staff at the AJB Service Center they would have heard about the 
steps that they routinely take to identify customer satisfaction and user feedback, 
particularly from job seeker customers. 

Findings an d Recommendations, page 5. p aragraph^ In the last sentence, there are two 
items that were overlooked. The term “casual” should be replaced by “anonymous” as 
was done in other parts of the paper, per our last round of comments and the “(who 
account for the majority of site visits)” should be deleted, as was done elsewhere, per our 
previous comments that this statement is unsubstantiated. 

Fi ndings and Recommendations, page 6, paragraph 2: As stated in comment number 2, 
changes in the number of hits usually reflect changes in how the web site is structured, 
rather than how often it is used. As the previous sentence gives a picture of site 
utilization, we would recommend deleting the “Conversely, since August 1999, the 
number of server accesses (“web hits”) has steadily declined,” as it adds no real value to 
the site utilization discussion, it only confuses it. 

Findings and Recommendations, page 6- paragraph 2: The second to last sentence in the 
paragraph states, “Nonetheless, ETA officials acknowledge that growth in usage of the 
AJB web site has not kept pace with the number of available positions.” Not only do 
officials not acknowledge this point, it is unclear what point this sentence is trying 
to make. It needs to be rephrased along with the last sentence which seems to be related 
to the previous sentence, as the point that you are attempting to make is unclear. 

Findings and Recommendations, page 6. Figure 2 : This graph should be deleted, per 
comment number 24, or changed to show AJB visits as discussed in the first sentence of 
the paragraph above it. 

Findings and Recommendations, p age 7. paragraph 4: You incorporated our comments 
and changed the sentence “in some cases, it may require navigating as many as six to 
eight screens before a specific job search result can be retrieved to AJB s information 
architecture currently requires four or five clicks, depending on the specificity of the 
search and the options chosen”, based on the AJB implementation of a new look and feel 
in a release that went live last December. However, you fail to mention that the AJB used 
the information in the evaluation and usability studies you referred to, to develop the new 
version and will be using them further to guide additional changes in the future. 

Findings and Recommendations, page 8, l ast paragraph; While it is true that “Job seekers 
visi ting a One-Stop have access to core services provided by ACK through the Internet.”, 
you still fail to mention that since AJB is on the Internet, many users can access and use 
the services without ever going to or registering with a One-Stop Center. If they do not 
have access to a computer at home, they do so through libraries, community centers, or 
churches. 



29. Findings and Recommendations, pa g e 9. first paragraph (after bullet list'): The last 
sentence states ETA should utilize WIA reporting requirements for the purpose of 
developing applicable outcome performance measures.” While these reporting 
requirements should be taken into account in developing applicable performance 
measures, we believe utilize is too strong a word. As pointed out before, self-service and 
information services are specifically exempted from the cited measures and applicable 
measures for these activities might be quite different from WIA reporting requirements. 

30 . Fi ndings and Recommendations, page 9. second paragraph (after bullet list!: The first 
sentence says “These performance measures certainly address some aspects of self- 
directed employment services, as they will primarily be provided through the One-Stop 
system.” We don’t know of any data that would validate this assertion. As per comment 
number 28, ACK/AJB services are available anywhere access to the Internet is available. 

31. Findings and Recommendations, page 10. second paragraph (after bullet list): The first 
sentence indicates that “previous evaluation of ‘customer satisfaction’ with regard to AJB 
has been completed, but job seeker satisfaction has played a limited role.” This is 
inaccurate as per our previous comments number 3 and 22. 

32. Findings and Recommendations, page 1 0. second paragraph (after bullet list): The third 
sentence indicates that “As a majority of the program’s impact is derived from job 
seekers...”. There is no data that would validate this assertion? in fact, many states would 
argue that AJB has as great (or even greater) an impact on the employer community. 

33 . Findings and Recommendations, page 1 1 . first paragraph: This paragraph restates the 
inaccurate information that was just discussed in die two previous comments. 

34 . Recommendations for Finding 1. page 12 . Recommendation 1 ; We do not agree with this 
recommendation, as per comment number 7. Doing an on-line survey is very expensive 
and tim e-consuming and we feel that there are better and less costly ways to gain this 
same type of information, namely focus groups, e-mail feedback and other means of user 
input. Again, it makes little sense to spend more money to determine a customer’s 
satisfaction with a service, than it does to provide the service itself. 

35 . R prr»mmendations for Finding 1 . page 12. Recommendation 2; As pointed out in our 
previous comments to your office, self-service systems are exempted from the core WIA 
performance measures. However, we agree with your recommendation as it has been 
rewritten and have already begun work on this recommendation, per comment number 8 . 

36 . Recommendations for Finding 1. page 12. Recommendation 3: We agree with this 
recommendation, per comment number 9, although it lacks specificity. We need to begin 
work on determining what type of customer satisfaction data is relevant to the electronic 
tools. 
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37. Finding 2. page 13. second bullet: The last part of the sentence says current procedures 
are “awkward and time consuming.” While this may be the case, there are limits to the 
availability of the database that contains the characteristic and demographic data, as it 
must be kept off-line to protect this sensitive information from hackers. As per comment 
number 5, we can work with the enforcement agencies to develop a better process, but 
they must realize that because this sensitive data has to be kept off-line, it will never be 
able to be retrieved at the point of the actual search (i.e., real-time) for security and 
privacy reasons. 

38. Finding 2. page 13. fourth bullet: Not entirely true; Babel Fish is available once again, 
per comment number 6 and 13. 

39. Compliance Requirements Related to AJB. page 16. bullet CO: Per comment numbers 4 
and 10, it is not appropriate for ETA or AJB to advise employers of an OFCCP regulatory 
requirement. Per the February 22, 2000 agreement, we agreed to provide a link to an 
OFCCP web page and label it “... to find out more about your responsibilities as a Federal 
Contractor, click here.” That link exists today in the employers’ registration process, and 
comes up whenever a registering employer self-identifies as a federal contractor. We 
believe this effectively addresses this issue. 

40. Compliance Requirements Related to AJB. page 1 6. bullet (31: The first sentence ends 
with “... and does not encourage users to provide pertinent information.” Per comment 
number 1 2, the language to “encourage” job seekers to provide the demographic 
infonnation was provided to AJB by the enforcement agencies. We are not opposed to 
changing the language if they provide us with language they think would better suit this 
purpose. 

4 1 . Compliance Requirements Related to AJB. page 16. bullet (31: “All the demographic 
information should be presented together and job seekers should be encouraged to 
provide the information.” It appears from this statement that you are saying the veterans 
and demographic infonnation should be asked for all in one place. As explained in 
comment 14, the veterans’ information is not currently part of the “electronic tear-off 
sheet” and is kept separate from the other information, as we are required by law to ask 
about veterans status and collect this information, but we do not have the authority to 
require the collection of demographic infonnation. The language used on the site to ask 
for demographic infonnation was provided by the DOL enforcement agencies, and was 
not drafted by ETA or AJB, per comment numbers 12 and 40. 

42. Findings and Recommendations, page 17. second paragraph: It should be noted that there 
are many other users of AJB, not just the United States Employment Service, as the 
paragraph seems to imply. 

43. Findings and Recommendations, page 17. fifth paragraph: It should be noted that while 
AJB is a federally-assisted program funded by U.S. DOL, it is not run by the U.S. DOL. 
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The grant is administered for the State of New York by the America’s Job Bank Service 
Center. 

Findings and Recommendati ons, pace 19. bullet ( 7 \. In the first sentence, there is a typo. 
One of the “ETA has completed the process of creating a demographic historical” should 
be deleted, as it is in the sentence twice. 

Recommendations for Finding 2. Recommendation 1: We believe we have met the spirit 
of this finding and it should be removed. See comments number 4, 1 0 and 39. 

Recommendations for Finding 2. Recommendation 2: We agree and have already begun 
to do so, per comment number 11. 

Recommendations fo r Finding 2. Recommendation 3: We agree, with the caveats 
mentioned previously in comments number 12 and 40. 

Recommendations for Findi ng 2, Recommendation 4: We believe we are already doing 
this, per comments number 6, 1 3 and 38. 



Recommendations for Fin ding 2. Recommendation 5: We will be impleme nting the sub- 
category in the September release of AJB, but it will not be a part of the “electronic tear- 
off sheet”, per comments number 14 and 41 . 

Recommendations fo r Finding 2. Recommendation 6: We agree to provide written 
instructions on the use of the historical database, per the caveat mentioned in comment 
number 1 5, that the written instructions are for enforcement agency distribution. 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




Efowllouol Resumes intormotioii Centei 



NOTICE 



Reproduction Basis 



This document is covered by a signed "Reproduction Release (Blanket)" 
form (on file within the ERIC system), encompassing all or classes of 
documents from its source organization and, therefore, does not require a 
"Specific Document" Release form. 



X 



This document is Federally- funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may be 
reproduced by ERIC without a signed Reproduction Release form (either 
"Specific Document" or "Blanket"). 



EFF-089 (1/2003) 



